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July  26, 1996 

The  Honorable  Norman  D.  Dicks 
House  of  Representatives 

Dear  Mr.  Dicks: 

In  response  to  your  request,  we  reviewed  the  Air  Force’s  reliability, 
maintainability,  and  availability  (RM&a)  evaluation  of  the  C-17  airlifter. 
Specifically,  we  reviewed  the  planning  and  preparation  for  the  evaluation, 
monitored  its  execution,  and  assessed  the  results  reported  by  the  Air 
Force.  In  addition,  as  you  requested,  we  obtained  information  on  the 
extent  that  the  evaluation  would  demonstrate  the  C-17’s  wartime  surge 
rate. 


Background 


I 


The  C-17  mUitaiy  transport,  which  is  being  produced  for  the  Air  Force  by 
the  McDonnell  Douglas  Corporation,  is  designed  to  airlift  substantial 
payloads  over  long  ranges  vdthout  refueling.  The  Air  Force  intends  the 
C-17  to  be  its  core  airlifter  and  the  cornerstone  of  its  future  airlift  force. 
The  Congress  had  appropriated  about  $20.7  billion  and  authorized  the 
acquisition  of  40  aircraft,  through  fiscal  year  1996,  for  the  C-17  program. 
The  $20.7  billion  includes  $6.9  billion  for  research  and  development, 
$14.6  billion  for  procurement,  and  $170  million  for  military  construction. 
The  Congress  has  also  authorized  the  Department  of  Defense  (dod)  to 
enter  into  a  multiyear  contract  for  the  acquisition  of  the  remaining 
80  aircraft  of  the  120  aircraft  C-17  program.  As  of  July  3, 1996,  27  aircraft 
have  been  delivered. 


The  C-17  development  contract  required  the  Air  Force  to  conduct  a  30-day 
evaluation  of  the  aircraft’s  compliance  with  rm&a  ^ecifications.  The 
evaluation  was  also  used  to  determine  how  much  of  a  $  12-million 
incentive  fee  the  contractor  was  entitled  to  for  meeting  those 
specifications.  In  October  1992,  the  Air  Force  developed  a  draft  rm&a 
evaluation  plan  that  was  closely  tailored  to  the  contract  specifications. 
The  plan  was  revised  during  1994  and  issued  in  July  1994. 


The  30-day  rm&a  evaluation  was  conducted  between  July  7  and  August  6, 
1995.  It  consisted  of  a  23-day  peacetime  segment  and  a  7-day  wartime 
segment.  Aircraft  operations,  using  12  aircraft,  were  conducted  at 
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6  U.S.  airfields  and  1  overseas  base.  Table  1  shows  the  number  of 
missions,  sorties,^  and  flight  hours  flown  during  the  evaluation. 


Table  1:  Missions,  Sorties,  and  Fiight 
Hours 

Segment 

Days 

Missions 

Sorties 

Flight  hours 

Peacetime 

23 

173 

334 

1,192 

Wartime 

7 

61 

179 

1,067 

Total 

30 

234 

513 

2,259 

Missions  included  logistics  (transporting  equipment,  personnel,  and 
supphes);  joint  operations  (training  with  equipment  and  personnel  from 
the  Army);  and  peacetime  aircrew  training.  The  wartime  logistics  missions 
were  designed  to  simulate  long-range  movement  of  equipment,  personnel, 
and  supplies  to  forward  operating  bases  or  small  austere  airfields. 
Peacetime  and  wartime  missions  included  aerial  refueling;  equipment  and 
personnel  airdrops;  formation  flying;  low-level  operations;  and  operations 
into  small,  austere  airfields.  The  wartime  missions  ranged  from  12.5  to 
26  hours,  wtule  the  peacetime  missions  ranged  from  2  to  20.5  hours. 

By  the  end  of  the  evaluation,  the  C-17  fleet  had  logged  about  13,000  total 
operational  flying  hours  since  initial  squadron  operations  began  in  1993. 
The  RM&A  evaluation  represents  about  2  percent  of  the  100,000  flying  hours 
needed  to  meet  aircraft  fleet  maturity. 


Results  in  Brief 


The  Air  Force  reported  that  the  C-17  met  or  exceeded  10  of  the  11  contract 
specification  requirements  during  its  rm&a  evaluation.  However,  the 
evaluation,  which  was  based  on  the  revised  plan  developed  in  1994,  was 
less  demanding  than  the  one  called  for  in  the  draft  1992  plan.  The  reduced 
rigor  stemmed  primarily  from  changes  in  the  nmnber  of  aircraft  sorties, 
average  sortie  length,  and  total  flying  hours.  These  changes  altered  the 
proportional  mix  of  sorties  to  flying  hours  that  had  been  developed  based 
on  contract  specifications.  The  altered  mix  of  sorties  to  flying  hours 
weakened  the  link  between  the  rm&a  evaluation  results  and  the 
measurement  criteria,  which  were  based  on  more  demanding  test  profiles. 
The  RM&A  evaluation  was  also  less  demanding  because  it  had  fewer 
airdrops  and  landings  at  small  austere  airfields  than  originally  planned  and 
flew  cargo  loads  that  were  significantly  lighter  than  projected  in  the 
contract  specifications. 


‘During  the  RM&A  evaluation,  a  sortie  began  with  the  takeoff  of  the  aircraft  and  ended  when  the 
aircraft  returned  to  the  ground  and  either  (1)  stopped  its  engines;  (2)  remained  on  the  ground  for  over 
6  minutes  (except  for  direct  delivery  and  tactical  proficiency  sorties);  or  (3)  changed  aircrews. 
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A  less  demanding  rm&a  evaluation  might  have  masked  reliability  and 
maintainability  problems  and  made  the  evaluation  a  less  reliable  source  of 
information  for  the  C-17  full-rate  production  decision.  However,  the 
results  of  the  Air  Force’s  3  years  of  operational  testing,  although  less 
impressive  than  the  results  of  the  rm&a  evaluation,  also  show  the  aircraft 
generally  met  rm&a  requirements  with  the  exception  of  those  related  to 
built-in-test  parameters. 

The  RM&A  evaluation  was  not  a  statistically  valid  test  for  determining  C-17 
fleet  wartime  utilization  rates.  Fleet  utilization  rates  refer  to  the  number  of 
hours  per  aircraft  that  a  fleet  of  airplanes  are  in  the  air  on  a  given  day.  The 
evaluation  did  not  demonstrate  or  prove  what  a  mature  C-17  fleet  would 
do  during  45  days  of  wartime  surge  operations.  It  simply  demonstrated 
that  a  high  utilization  rate  could  be  achieved  over  a  48-hour  period. 

Finally,  in  awarding  the  incentive  fee,  the  Air  Force  credited  the  C-17 
aircraft  with  meeting  the  full  mission  capable  rate  goal.  However,  during 
the  RM&A  evaluation,  the  aircraft  was  restricted  from  performing  formation 
personnel  airdrop  under  operationally  representative  conditions  and  was 
rated  not  functionally  effective  for  aeromedical  evacuation.  As  a  result,  the 
$5.91-nuIlion  incentive  fee  was  $750,000  higher  than  justified. 


The  C-17  met  or  exceeded  10  of  the  11  rm&a  evaluation  contract 
specification  requirements.  (See  app.  I.)  However,  the  rm&a  evaluation  was 
less  demanding  than  originally  called  for  in  the  contract  i^ecifications  and 
the  1992  draft  rm&a  plan.  The  rm&a  evaluation,  based  on  the  1994  revised 
plan,  decreased  the  ratio  of  sorties  to  total  flying  hours.  The  decrease 
weakened  the  link  between  the  evaluation  as  executed  and  the  rm&a 
measurement  criteria.  In  addition,  the  evaluation  was  less  demanding 
because  the  number  of  airdrops  and  landings  on  small  austere  airfields 
was  decreased  and  lighter  average  cargo  loads  than  called  for  in  the 
contract  specifications  were  carried. 


RM&A  Evaluation 
Was  Less  Demanding 
Than  Originally 
Planned 


Revised  Mission  Profiles 
Reduce  the  Ratio  of  Sorties 
to  Flying  Hours 


The  1992  draft  rm&a  evaluation  plan  was  based  on  25  C-17  mission  profiles 
representing  the  aircraft’s  projected  peacetime  and  wartime  usage  over  a 
30,000-hour  airframe  life  included  in  the  development  contract  In 
developing  its  1992  draft  plan,  the  Air  Force  conducted  extensive  analyses 
and  reviews  to  ensure  that  the  plan  adhered  to  the  contractual 
requirements. 
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In  January  1994,  as  part  of  the  settlement  related  to  the  C-17  development 
program  between  DOD  and  the  contractor,  dod  directed  the  Air  Force  and 
the  contractor  to  revise  the  C-17  rm&a  evaluation  plan  to  make  it  more 
operationally  realistic.  That  is,  to  more  realistically  mirror  the  planned  use 
of  the  aircraft.  In  addition,  because  of  rehability  problems  with  the  C-17, 
the  scheduled  November  1995  Defense  Acquisition  Board  was  to  consider 
the  evaluation  results  when  it  decided  whether  to  continue  the  C-17 
program  beyond  40  aircraft.  As  part  of  the  1994  revisions,  the  Air  Mobility 
Command  changed  the  mission  profiles  used  in  the  October  1992  draft 
plan  because  they  did  not  represent  complete  and  comprehensive 
missions.  Command  officials  were  also  concerned  that  the  1992  draft  plan 
would  not  demonstrate  the  aircraft’s  wartime  surge  utilization  rates 
included  in  the  C-17  Operational  Requirements  Document. 

In  July  1994,  the  Air  Force  issued  the  revised  rm&a  evaluation  plan.  The 
plan  included  a  wartime  scenario  representative  of  a  major  regional 
contingency,  additional  sorties  to  simulate  complete  missions,  and 
additional  flying  hours  to  increase  the  aircraft’s  utilization  rate. 

The  revised  mission  profiles  in  the  final  rm&a  plan  increased  the  total 
number  of  flying  hours,  number  of  aircraft  sorties,  and  average  wartime 
sortie  duration,  but  did  not  maintain  the  proportional  mix  of  sorties  to 
flying  hours  that  was  based  on  contract  specifications.  'The  impact  of  these 
changes  was  longer  duration  wartime  sorties  and  a  reduced  ratio  of  sorties 
to  flying  hours,  resulting  in  less  stress  on  the  rm&a  aircraft  than  originally 
planned.  Longer  missions  with  fewer  cycles,^  such  as  strategic  intertheater 
missions,  place  less  stress  on  an  aircraft  and  will  result  in  longer  aircraft 
hfe.  The  1992  draft  evaluation  plan  provided  for  1,725  total  fljdng  hours. 
'The  rm&a  evaluation  increased  the  level  to  2,259  flying  hours,  an  increase 
of  31  percent  over  the  draft  plan.  The  total  number  of  sorties  flown 
increased  12  percent,  but  the  average  sortie  time  increased  17  percent. 
Peacetime  sorties  increased  34  percent,  from  248  to  334,  but  the  number  of 
wartime  sorties  decreased  15  percent,  from  211  to  179.  Although  the 
change  in  the  duration  of  the  average  peacetime  sortie  was  negligible,  the 
average  wartime  sortie  increased  by  50  percent,  fi:om  3.99  to  5.97  hours. 
(See  £^p.  n.) 

Because  the  average  wartime  sortie  significantly  increased,  the  number  of 
sorties  in  relation  to  the  number  of  flying  hours  was  less  than  planned  in 
the  1992  draft  rm&a  evaluation  plan.  We  estimate  that  if  the  average 
duration  of  peacetime  and  wartime  aircraft  sorties  had  not  changed,  the 


^An  aircraft  cycle  is  defined  as  the  operation  of  an  aircraft  from  engine  start  to  engine  stop. 
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Air  Force  would  have  needed  to  fly  90  additional  sorties.  This  represents  a 
15-percent  increase  in  the  number  of  sorties  necessary  to  maintain  the 
proportional  mix  of  flying  hours  to  aircraft  sorties  identified  in  the  1992 
draft  evaluation  plan.  (See  app.  HI.) 


Changes  to  Mission 
Profiles  Weaken 
Application  of  RM&A 
Measurement  Criteria 


The  ratio  of  flying  hours  to  sorties  specified  in  the  contract  and  1992  draft 
plan  mission  profiles  was  based  on  the  profiles  used  in  the  development  of 
selected  rm&a  measurement  criteria  In  addition  to  reducing  the  stress  on 
the  RM&A  aircraft,  changes  to  the  original  mission  profiles  weakened  the 
hnk  between  the  rm&a  evaluation  scenarios  and  the  assessment  criteria 
developed  using  the  original  profiles. 

The  Air  Force  used  the  C-17  lifetime  mission  profiles  in  the  contract 
specifications  to  develop  the  test  profiles  in  the  1992  draft  of  the  rm&a 
evaluation  plan.  These  lifetime  mission  profiles  were  also  used  to  develop 
a  number  of  the  C-17  rm&a  growth  curve  parameters,^  such  as  mission 
completion  success  probability,  full  mission  capable  rate,  and  partial 
mission  capable  rate.  The  rm&a  growth  curves,  based  on  total  C-17  fleet 
flying  hours,  are  the  criteria  used  to  measure  the  C-17  rm&a  results. 

A  1981  report  by  the  contractor  noted  that  the  operational  profiles  flown 
during  the  rm&a  evaluation  must  be  the  same  as  the  profiles  used  to 
develop  the  growth  curves.  Since  the  original  mission  profiles  were  used 
as  a  basis  for  developing  rm&a  growth  curve  criteria,  a  revision  in  the 
profiles  required  a  corresponding  adjustment  in  the  respective  growth 
curves.  The  failure  to  make  such  an  adjustment  affected  the  use  of  the 
growth  curves  as  rm&a  measurement  criteria. 


Airdrops  and  Austere 
Airfield  Landings  Were 
Reduced 


The  total  number  of  airdrops  and  austere  airfield  landings  accomplished  in 
the  rm&a  evaluation  were  less  than  called  for  in  the  1992  draft  plan,  thus 
causing  less  stress  and  wear  on  the  C-17  aircraft  and  its  subsystems. 

•  The  total  number  of  airdrops  was  reduced  from  189  to  158,  a  16-percent 
reduction.  Wartime  airdrops  were  decreased  by  92  percent,  from  50  to  4. 
Air  Force  officials  stated  that  they  significantly  decreased  the  number  of 
wartime  airdrops  because  the  1992  Mobility  Requirements  Study  and  the 
1995  MobiUty  Requirements  Study  Bottom-Up  Review  Update  did  not 


^Growth  curves  represent  the  expected  behavior  of  the  fleet  for  selected  RM&A  parameters  at  a 
particular  point  in  the  maturity  of  the  aircraft.  Appendix  I  contains  a  listing  of  all  RM&A  evaluation 
measurement  parameters. 
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include  airdrop  as  a  requirement  for  a  major  regional  contingency 
warfighting  scenario. 

•  The  number  of  C-17  small,  austere  airfield  landings  was  16  percent  less 
than  called  for  in  the  1992  draft  plan — 138  instead  of  164.  According  to  Air 
Mobility  Command  officials,  they  reduced  the  number  of  landings  from 
164  to  148  because  they  did  not  believe  the  additional  landings  were 
needed  to  determine  the  rm&a  evaluation  impact  and  an  additional 
10  planned  landings  were  not  accomplished  due  to  mechanical  or 
environmental  problems. 


Although  the  type  of  cargo^  carried  during  the  rm&a  evaluation  was 
realistic,  the  avers^e  weight  of  the  loads  was  less  than  half  that  projected 
in  the  mission  profiles  in  the  contract  specifications.  As  a  result,  the 
aircraft  and  its  subsystems  experienced  less  stress  and  wear  during  the 
evaluation. 

Based  on  the  mission  profiles  in  the  contract  specifications,  the  average 
cargo  weight  per  mission  over  the  lifetime  of  the  C-17  aircraft  is  48,649 
pounds.  However,  the  aircraft  only  carried  an  average  cargo  weight  of 
approximately  23,000  pounds  during  the  rm&a  evaluation.  In  addition,  the 
actual  average  cargo  weight  carried  during  landings  on  small  austere 
airfields  was  nearly  2.5  times  less  than  the  average  cargo  loads  projected 
in  the  contract  specifications  (about  18,600  rather  than  46,000  pounds).  We 
are  currently  reviewing  the  C-17’s  performance  in  Bosnia  This  work 
should  provide  greater  insights  into  aircraft  performance  when  carrying 
heavier  loads. 


Average  Cargo  Loads  Less 
Than  Planned 


Extent  to  Which 
Utilization  Rates  Were 
Demonstrated 


One  reason  for  revising  the  1992  draft  rm&a  evaluation  plan  was  to 
demonstrate  the  wartime  surge  utilization  rate  included  in  the  C-17 
Operational  Requirements  Document — that  is,  operate  15.2  flying  hours  a 
day  per  aircraft  for  45  days.  Aircraft  utilization  rate  goals  were  met  and 
slightly  exceeded  during  the  rm&a  evaluation.  However,  the  evaluation  was 
not  intended  to  provide  a  statistically  valid  basis  for  predicting  the  C-17’s 
abUity  to  meet  its  wartime  surge  rate.  It  did  not  demonstrate  what  a 
mature  C-17  fleet  would  do  during  45  days  of  wartime  surge  operations. 
The  evaluation  simply  demonstrated  that  high  utilization  rates  could  be 
achieved  over  a  48-hour  period. 


■•rm&a  evaluation  cargo  included  oversized  and  outsized  equipment,  such  as  6-ton  trucks,  Bradley 
fighting  vehicles,  and  tanks  as  well  as  palletized  bulk  cargo. 
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The  actual  peacetime  utilization  rate  was  4.3  hours  per  aircraft.  The 
wartime  sustained  rate  was  12.7  hours,  with  wartime  surge  rates  of  16.6 
and  17. 1  hours  demonstrated  during  two  24-hour  periods.  According  to 
DOD  and  Air  Force  officials,  it  would  not  be  economically  feasible  to 
conduct  more  realistic  tests  because  of  the  large  amount  of  flying  hours 
and  resources  required. 

Moreover,  while  utilization  rates  are  used  as  one  basis  for  budgeting  for 
logistics  resources  and  mission  planning,  a  higher  utilization  rate  does  not 
necessarily  mean  that  one  aircr^t  is  a  better  airlifter  than  another.  Simply 
stated,  utilization  rate  is  the  number  of  hoins,  per  aircraft,  that  a  fleet  of 
airplanes  is  in  the  air  on  a  given  day.  More  time  in  the  air  yields  higher 
utilization  rates,  more  time  spent  on  the  ground  yields  lower  utilization 
rates.  The  rate  is  a  function  of  the  total  airlift  system  that  includes,  among 
other  things,  aircraft,  personnel,  airfields,  logistics  resources,  and 
concepts  of  operation.  All  these  factors  influence  the  attainment  of  a 
utilization  rate  objective,  and  most  have  little  or  nothing  to  do  with  an 
aircraft’s  inherent  capability.  For  example,  utilization  rates  can  be 
increased  by  longer  mission  flying  times,  slower  airspeeds,  aircrew 
augmentation,  and  ramp  space  availability.  Conversely,  a  faster  aircraft 
flying  the  same  distance  will  have  a  lower  utilization  rate. 


Incentive  Fee  Award 
Overstated  Given 
'  Equipment  and 
Capability  Exclusions 


The  Air  Force  awarded  the  C-17  contractor  $5.91  million  of  the  maximum 
$  12-million  incentive  fee.  However,  our  review  showed  that  amount  was 
$750,000  more  than  justified  under  the  contract.  The  amount  should  have 
been  reduced  because  the  C-17  aircraft  were  not  full  mission  capable 
during  the  evaluation.  (See  app.  IV  for  our  calculation  of  the  appropriate 
incentive  fee.) 


Aircraft  Could  Not  Perform 
Formation  Personnel 
Airdrop  or  Aeromedical 
Evacuation  Missions 


According  to  the  C-17  development  contract,  the  rm&a  incentive  fee  was  to 
be  based  on  the  degree  that  the  contractor  met  each  of  11  individual  rm&a 
parameter  goals.  That  is,  to  receive  the  total  $12-million  payment,  the 
contractor  had  to  achieve  the  goals  for  each  of  the  11  parameters.  If  any 
parameter  was  not  met,  the  payment  was  reduced  by  the  amount  for  that 
parameter  and  half  of  the  amounts  for  the  remaining  parameters.  The 
contractor  was  awarded  only  $5.91  million  because  the  C-17  did  not  meet 
the  requirement  for  the  built-in-test  false  indication  parameter. 


In  awarding  the  $5.91-miUion  fee,  the  Air  Force  gave  the  contractor  credit 
for  meeting  the  full  mission  capable  goal.  In  our  opinion,  none  of  the 
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aircraft  should  have  been  considered  full  mission  c£Q)able  during  the 
evaluation.  First,  the  Air  Force,  based  on  the  results  of  developmental 
testing,  had  restricted  the  aircraft  from  executing  the  formation  personnel 
airdrop  mission  under  operational  conditions  for  safety  reasons.  This 
mission  was  a  requirement  identified  in  C-17  operational  documents.  The 
restriction  on  formation  personnel  airdrop  existed  because  turbulence 
caused  by  the  aircraft  can  cause  injuries  to  paratroopers.  As  a  result,  the 
aircraft  are  not  permitted  to  fly  in  sufficiently  close  formation  to  airdrop 
the  required  number  of  personnel  under  operationally  representative 
conditions  as  required  by  the  contract  specification. 

Second,  the  aircraft  were  not  considered  effective  for  the  aeromedical 
evacuation  mission,  which  was  not  completely  tested  during  the  rm&a 
evaluation.  The  aircraft  were  reconfigured  to  demonstrate  this  capability, 
but  not  all  the  systems  that  would  be  needed  to  accompUsh  the  mission 
were  used.  Initial  operational  test  and  evaluation  testing,  which  included 
the  information  developed  during  the  rm&a  evaluation,  identified  a  number 
of  problems  that  prevented  the  aircraft  fi-om  being  considered  able  to 
perform  the  aeromedical  evacuation  mission.  For  example,  the  emergency 
oxygen  supply  to  patient  litters  was  defective.  As  a  result,  the  Army  has 
classified  the  aircraft  as  not  functionally  effective  for  aeromedical 
evacuation. 


Recommendation 


We  recommend  that  the  Secretary  of  Defense  direct  the  Secretary  of  the 
Air  Force  to  initiate  action  to  recover  the  $750,000  in  incentive  fee 
overpasment  from  the  contractor. 


Agency  Comments 
and  Our  Evaluation 


In  commenting  on  a  draft  of  this  report,  dod  partially  concurred  with  our 
findings  but  did  not  concur  with  our  recommendation,  dod  stated  that  the 
1994  plan  was  actually  more  extensive  and  more  operationally 
representative  than  the  draft  1992  plan  because  it  increased  the  total  flying 
hours,  the  number  of  sorties,  wartime  sortie  duration,  aerial  refueling,  and 
formation  flying  missions.  However,  dod  acknowledged  that  the  1994  plan 
reflected  (1)  a  30-percent  reduction  in  the  number  of  airdrop  sorties,  (2)  a 
10-percent  reduction  in  the  number  of  small  austere  airfield  landings,  and 
(3)  more  than  a  60-percent  reduction  in  the  average  cargo  loads  carried 
during  the  evaluation  compared  to  the  1992  draft  plan,  dod  indicated  that 
the  1992  draft  plan  should  not  be  used  as  a  benchmark,  rather  the  contract 
specification,  including  the  1994  Settlement  Agreement  between  dod  and 
the  contractor,  should  have  been  used.  Fluther,  dod  stated  that  not 
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a(^justing  the  growth  curves  to  account  for  the  changes  made  in  the  plan 
would  have  had  only  minimal  impact  on  the  results  of  the  evaluation. 

We  did  not  use  the  1992  draft  plan  as  a  benchmark.  Rather,  we  pointed  out 
that  scenarios  in  the  1992  draft  plan  and  the  growth  curves,  which  are  the 
criteria  used  to  measure  the  success  of  the  evaluation,  were  both  based  on 
the  same  factors  from  the  contract  specification.  The  1994  plan  changed 
the  scenarios  being  flown  in  the  evaluation  to  make  it  more  operationally 
realistic  and  to  more  closely  resemble  a  major  regional  contingency. 
However,  the  growth  curves  were  not  adjusted,  dod  provided  no 
documentation  to  support  its  assertion  that  ac^justing  the  growth  curves 
would  have  had  only  minimal  impact.  Moreover,  the  C-17  contractor  has 
stressed  that  the  profiles  flown  during  the  evaluation  must  be  the  same  as 
those  used  to  develop  the  growth  curves. 

DOD  acknowledged  that  the  limited  wartime  surge  activities  during  the 
RM&A  evaluation  did  not  provide  a  statistical  basis  for  predicting  the  C-17’s 
ability  to  meet  its  wartime  surge  rate,  dod  said  we  questioned  the  value  of 
utilization  rates  in  this  report,  even  though  in  a  prior  report  we  had 
indicated  that  utilization  rates  were  a  useful  statistic  when  comparing 
aircraft.  Our  point  in  the  prior  report  was  that  the  value  of  comparing 
utilization  rates  was  undermined  when  dod  artificially  constrained  the 
utilization  rate  of  one  aircraft  while  using  the  planned  wartime  surge 
utilization  rate  for  another.  However,  to  assure  that  our  position  in  this 
report  is  clear,  we  have  modified  the  text  dealing  with  utilization  rates. 

DOD  disagreed  with  our  recommendation  to  seek  reimbursement  of 
$750,000  ft-om  the  contractor,  asserting  that  the  aircraft  was  properly 
considered  full  mission  capable  as  long  as  aU  the  equipment  required  for 
the  mission  was  available  and  operative.  The  contract  specification  defines 
full  mission  capable  as  the  aircraft  being  capable  of  performing  all  of  its 
design  missions.  Since  it  could  not  perform  the  formation  personnel 
airdrop  and  aeromedical  evacuation  missions,  we  believe  that  the  aircraft 
was  incorrectly  listed  as  full  mission  capable.  The  aircraft  is  restricted 
from  performing  the  formation  personnel  airdrop  mission  for  safety 
reasons.  While  the  aircraft  were  reconfigured  to  perform  the  aeromedical 
evacuation  mission,  some  of  the  equipment  was  not  tested  to  ensxire  it  was 
operating  as  needed  to  enable  the  aircraft  to  perform  the  mission.  Further, 
the  aircraft  was  classified  as  not  functionally  effective  for  aeromedical 
evacuation  as  a  result  of  initial  operational  test  and  evaluation  testing 
because  of  a  number  of  problems,  including  equipment  problems. 
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We,  therefore,  continue  to  believe  that  the  aircraft  should  not  have  been 
considered  as  full  mission  capable  and  the  contractor  should  not  have 
been  paid  the  incentive  award  fee  of  $750,000  for  meeting  the  full  mission 
enable  objective. 


Scope  and 
Methodology 


To  determine  the  overall  performance  of  the  C-17  during  the  evaluation, 
we  monitored  the  conduct  and  coordination  of  the  rm&a  evaluation  from 
the  437th  Airlift  Wing,  Charleston  Air  Force  Base,  South  Carolina  This 
included  the  daily  rm&a  evaluation  activities  of  the  exercise  as  well  as 
related  data  collection  and  documentation  activities.  We  also  flew  on 
selected  C-17  missions  and  observed  ground  operations  at  C-17  operating 
bases,  including  North  Auxiliary  Airfield,  South  Carolina;  Pope  Air  Force 
Base,  North  Carolina;  and  forward  operating  bases  at  Barstow-Daggett 
Municipal  Airport,  California,  and  Bicycle  Lake  Army  Airfield,  CaUfomia. 

To  determine  the  validity  of  the  test  design,  mission  mix,  and  operational 
realism  of  the  exercise,  we  analyzed  the  rm&a  evaluation  plan.  Specifically, 
we  reviewed  its  piupose,  structure,  preparation,  and  execution  as  weU  as 
the  results  of  the  evaluation.  We  also  interviewed  officials  firom  the  14th 
Airlift  Squadron;  the  17th  Airlift  Squadron;  the  437th  Airlift  Wing;  the  315th 
Reserve  Airlift  Wing;  Air  Mobility  Command  Headquarters;  the  C-17 
System  Program  Office;  C-17  Site  Activation  Task  Force;  San  Antonio  Air 
Logistics  Center,  Air  Force  Operational  Test  and  Evaluation  Center;  Air 
Force  Office  of  Operational  Test  and  Evaluation;  Headquarters  U.S.  Air 
Force;  U.S.  Army  Test  and  Evaluation  Command;  U.S.  Army  Test  and 
Experimentation  Command;  Institute  for  Defense  Analysis;  and  McDonnell 
Douglas,  the  C-17  contractor. 

We  conducted  our  review  from  June  1995  to  March  1996  in  accordance 
with  generally  accepted  government  auditing  standards. 


We  are  sending  copies  of  this  report  to  the  Chairmen  and  Ranking 
Minority  Members  of  the  Senate  Committee  on  Armed  Services;  the 
Subcommittee  on  Defense,  Senate  Committee  on  Appropriations;  the 
House  Committee  on  National  Security;  the  Subcommittee  on  National 
Security,  House  Committee  on  Appropriations;  the  Secretaries  of  Defense 
and  the  Air  Force;  and  the  Director  of  the  Office  of  Management  and 
Budget.  We  will  also  provide  copies  to  other  interested  parties  as 
requested. 
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If  you  or  your  staff  have  any  questions  concerning  this  report,  please 
contact  me  on  (202)  5124841.  The  major  contributors  to  this  report  are 
listed  in  appendix  VI. 

Sincerely  yours, 


Louis  J.  Rodrigues 

Director,  Defense  Acquisition  Issues 
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Comparison  of  C-17  30-Day  RM&A 
Evaluation  With  Results  of  Operational  Test 
and  Evaluation 


Parameter 

Operational  test  and  evaluation  June  10, 1993,  to 
RM&A  evaluation  July  7  to  August  5, 1 995  August  5, 1 995 

Good 

Requirement 

Result 

Met 

Requirement 

Result 

Met 

MC  Rate  (%) 

t 

80.7 

90.6 

Yes 

80.8 

86.0 

Yes 

FMC  Rate  (%) 

t 

72.9 

85.1 

Yes 

73.0 

75.9 

Yes 

MCSP  (%) 

t 

85.8 

97.8 

Yes 

85.0 

92.0 

Yes 

MTBM  (l)(FH) 

t 

1.3 

3.4 

Yes 

1.3 

2.0 

Yes 

MTBM  (C)(FH) 

T 

0.6 

1.6 

Yes 

0.6 

0.9 

Yes 

MTBR  (FH) 

t 

2.2 

7.5 

Yes 

2.0 

6.3 

Yes 

MMH/FH  (MH) 

i 

28.4 

4.3 

Yes 

28.9 

5.4 

Yes 

MMTR  (MH) 

i 

8.2 

2.7 

Yes 

8.2 

3.0 

Yes 

BIT-D  (%) 

T 

95.0 

98.6 

Yes 

BiT-l  (%) 

t 

90.0 

95.2 

Yes 

BIT-Fl  (%) 

i 

5.0 

59.9 

No 

Legend 


MC  •  Mission  capable  (capable  to  perform  at  least  one  mission) 

FMC  Full  mission  capable  (capable  to  perform  all  missions) 

MCSP  Mission  completion  success  probability  (complete  mission  objectives  without 

experiencing  failure  or  performance  degradation  due  to  equipment  problems) 

MTBM(i)  Mean  time  between  maintenance-inherent  (mean  flight  hours  between  unscheduled, 

on-equipment,  inherent  maintenance  actions) 

MTBM(c)  Mean  time  between  maintenance-corrective  (mean  flight  hours  between 

unscheduled  corrective  actions) 

MTBR  Mean  time  between  removal  (mean  flying  hours  between  removal  of  any  repairable 

equipment) 

MMH/FH  Maintenance  man  hours  per  flying  hour  (total  maintenance  hours  expended  tor  each 

flight  hour) 

MMT  Mean  man  hours  to  repair  (the  mean  maintenance  man  hours  required  to  complete 

a  corrective  maintenance  action) 

BIT-D  Built-in-test  fault  detection  (percentage  of  occurrences  in  which  BIT  correctly 

detects  a  malfunction) 

BiT-j  Built-in-test  fault  isolation  (percentage  of  occurrences  in  which  BIT  correctly  isolates 

a  detected  malfunction  to  the  failed  equipment  item) 

BiT-FI  Built-in-test  false  fault  indication  (percentage  of  occurrences  in  which  BIT  indicated 

a  malfunction  when  none  existed) 
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Comparision  of  Flight  Hours,  Sorties,  and 
Average  Sortie  Time  Between  Original 
RM&A  Plan,  Revised  Plan,  and  Actual 


Revised  plan  Difference  between  Difference  between  actual 

plan  (July  (October  revised  and  original _ and  original 


Actual 

1994) 

1992) 

Number 

Percent 

Number 

Percent 

Flying  hours 

Peacetime 

1,191 

1,123 

884 

239 

27 

307 

35 

Wartime 

1,068 

1,029 

841 

188 

22 

227 

27 

Total 

2,259 

2,152 

1,725 

427 

25 

534 

31 

Sorties 

Peacetime 

334 

331 

248 

83 

33 

86 

35 

Wartime 

179 

175 

211 

(36) 

(17) 

(32) 

(15) 

Total 

513 

506 

459 

47 

10 

54 

12 

Average  sortie  time  (hours) 

Peacetime 

3.57 

3.39 

3.56 

(0.16) 

(05) 

0.02 

01 

Wartime 

5.97 

5.88 

3.99 

1.89 

47 

1.98 

50 

Total 

4.40 

4.25 

3.76 

0.49 

13 

0.64 

17 
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Comparison  of  Actual  RM&A  C-17  Aircraft 
Cycles/Sorties  With  Estimated 
Cycles/Sorties  Using  Average  Sortie  Times 
Contained  in  Original  Plan  _ 


Peacetime 

Wartime 

Total 

Flight  hours  original  plan  October  1992 

884 

841 

1,725 

Increase  from  original  to  actual 

307 

227 

534 

Actual  hours  flown 

1,191 

1,068 

2,259 

Actual  hours  divided  by 

Original  planned  average  aircraft  sortie 
time 

3.56 

3,99 

Equals 

Estimated  number  of  aircraft  sorties 
required 

335 

268 

603 

Actual  number  of  aircraft  sorties 

334 

179 

513 

Estimated  shortage  of  aircraft  sorties 

1 

89 

90 

Percent  shortfall 

0 

33 

15 
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RM&A  Evaluation  Incentive  Award  Fee 
Calculation 


Dollars  in  millions 

Parameter 

Maximumfee 

Amount 

awarded^ 

GAO 

assessment 

MCrate 

$1,500 

$0,750 

$0,750 

FMC  rate 

1.500 

0.750 

O'’ 

MCSP 

1.350 

0.675 

0.675 

MTBM  (1) 

0.900 

0.450 

0.450 

MTBM  (C) 

0.900 

0.450 

0.450 

MTBR 

1.350 

0.675 

0.675 

MMH/FH 

1.800 

0.900 

0.900 

MMTR 

1.800 

0.900 

0.900 

BIT-D 

0.360 

0.180 

0.180 

BIT-1 

0.360 

0.180 

0.180 

BIT-FI 

0.180 

0 

0 

Total 

$12,000 

$5,910 

$5,160 

Overpayment  ^ 

$0,750 

®The  contractor  failed  to  pass  the  requirement  for  built-in-test  false  indication  and,  therefore, 
received  no  award  for  this  parameter  and  only  half  the  maximum  fee  for  the  other  parameters, 
according  to  the  terms  of  the  contract. 

^The  contractor  did  not  pass  the  FMC  parameter  because  none  of  the  test  aircraft  were  full 
mission  capable;  therefore,  the  contractor  should  not  have  received  the  $750,000  incentive  fee. 
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Comments  From  the  Department  of  Defense 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-3000 


5om 


ACQUtSfTION  AND 
TECHNOUXSY 


.1  J  Jl'i'i 


Mr.  Louis  J.  Rodrigues 
Director,  Defense  Acquisition  Issues 
National  Security  and 
International  Affairs  Division 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 


Dear  Mr.  Rodrigues: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Acc<^ting  Office 
(GAO)  draft  report,  “C-17  AIRCRAFT:  RM&A  Evaluation  Less  Demanding  Than  Initially 
Planned,"  datedMayS,  1996  {GAO  Code  707  111),  OSD  Case  1147.  Tlie  Department  partially 
concurs  with  the  findings  in  the  report,  but  does  not  concur  with  the  lecommendation. 

The  DoD  stresses  that  the  primary  purpose  of  the  C- 1 7  Reliability,  Maintainability  and 
AvailabiUty  Evaluation  (RM&AE)  was  to  provide  a  demonstration  of  the  C-17  pto^  s 
compHance  with  the  program’s  contractual  RM&A  specification  requirements  It  does  not  a^e 
with  the  GAO  assertion  that  the  C-17  RM&AE  was  less  demanding  than  imtially  planned  (in  the 
draft  1992  RM&AE  plan)  or  that  it  may  have  masked  reliabiUty  and  maintainability 
The  RM&AE  conducted  between  July  and  August  1995  in  accordance  with  the  approved 
RM&AE  plan  was  actually  more  extensive  and  operationally  representative  than  what  was 
initially  reflected  in  the  draft  1992  plan,  which  the  GAO  incorrectly  uses  as  a  benchmark.  The 
contract  specification,  to  include  the  1994  Settlement  Agreement  between  tlw  DoD 
McDonnell  Douglas,  is  the  benchmark.  The  1994  RM&AE  plan  increased  the  total  flying  hours, 
the  number  of  sorties,  wartime  sortie  durations,  aerial  refuelings,  and  formation  flying  inissions. 

It  was  also  structured  to  assess  the  weapons  system’s  ability  to  achieve  its  peaceUme  and  w  ime 
utilization  rate  requirements. 

The  Department  does  not  share  the  GAO  view  that  increasing  the  total  number  of  Hying 
hours  the  number  of  aircraft  sorties,  and  the  average  wartime  sortie  durauon  resul^  in  less 
stress’being  placed  on  the  aircraft  during  the  RM&AE,  Due  to  the  wear  and  tear  placed  on 
aircraft  during  take-offs  and  landings,  the  additional  86  peacetime  sorties  flown  during  the 
RM&AE  certainly  placed  more  stress  on  the  aircraft  and  logistics  p  ^ . 

planned  While  32  (15  percent)  fewer  wartime  sorties  were  conducted  during  the  RM&AE,  the 
Department  notes  that  increasing  the  average  sortie  length  during  the  wartime  segmrat  m^e  the 
RM&AE  more  difficult  from  a  logistics  perspective,  since  ground  times  were  reduced  and 
personnel  had  less  time  to  service  and  repair  aircraft  between  sorties. 
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The  Department  agrees  that  the  RM&AE  reflected  a  30  percent  reduction  in  the  number 
of  airdrop  sorties,  a  10  percent  reduction  in  the  number  of  small  austere  airfield  landings,  and  a 
reduction  by  more  than  half  in  the  average  cargo  loads  carried  during  RM&AE,  compared  to 
those  identified  in  the  draft  1992  RM&AE  plan.  However,  those  changes  were  made  to  make  the 
RM&AE  more  operationally  representative.  For  example,  while  the  number  of  airdrop  sorties 
was  primarily  decreased  because  none  were  required  in  the  Department’s  1995  Mobility 
Requirements  Study  -  Bottom-up  Review  Update,  the  number  of  formation  sorties  flown  during 
the  RM&AE  was  increased  by  55  percent  to  better  reflect  the  current  tactics  for  the  employment 
of  the  aircraft.  While  the  Department  agrees  that  the  C-I7  aircraft  was  not  always  flown  with  full 
loads,  there  was  no  contractual  RM&AE  requirement  to  cany  any  operational  cargo  during  the 
evaluation.  However,  to  insure  operational  realism,  the  C-I7  aircraft  did  cany  a  spectrum  of 
operationally  representative  cargo  loads  (i.e.,  12  MlAI  Abrams  tanks  (125,000  lbs  each),  12 
Bradley  fighting  vehicles,  14  Sheridan  tanks  and  6034  personnel  into  seven  different  airfields 
(including  one  overseas  base))  during  the  RM&AE.  Overall,  the  C-17  aircraft  carried  10,723,1 19 
pounds  of  cargo  (10,031,469  lbs  airland/691,650  lbs  airdropped)  during  the  RM&AE.  It  should 
be  noted  that  some  types  of  missions,  such  as  pilot  proficiency  training  and 
positioning/depositioning  legs,  do  not  normally  cany  cargo. 

The  DoD  does  not  concur  that  failure  to  adjust  the  growth  curves  because  of  changes  in 
mission  profiles  affected  their  use  as  RM&A  measurement  criteria.  Any  adjustments  would  have 
had  minimal  impact  and  are  not  an  issue  since  the  0-17  far  surpassed  both  growth  curve  and 
mature  C-17  fleet  (100,000  fleet  flying  hours)  requirements.  As  illustrated  in  the  subject  GAO 
report,  the  results  from  the  RM&AE  in  1995  and  the  operational  test  and  evaluation  results 
between  July  1993  and  August  1995  clearly  show  that  the  C-17  far  surpassed  growth  curve 
values,  as  well  as  the  Operational  Requirements  Document  (ORD)  mature  fleet  requirements. 

The  Department  would  note  that  while  the  use  of  growth  curves  as  a  management  tool  has 
proven  effective  on  this  and  other  programs,  the  critical  consideration  is  whether  the  system  has 
met  its  operational  requirements  as  reflected  in  the  ORD. 

The  Department  concurs  that  limited  wartime  surge  activities  during  the  RM&AE  did  not 
provide  a  statistically  valid  basis  for  predicting  the  C-17’s  ability  to  meet  its  wartime  surge  rate. 
However,  the  intent  was  only  to  illustrate  that  the  high  utilization  rate  could  be  attained  in  the 
field  under  operationally  representative  conditions.  Providing  a  statistically  significant  basis  for 
showing  that  the  utilization  rate  could  be  maintained  for  extended  periods  would  have  been  cost 
prohibitive.  Therefore,  the  Department  undertook  a  ground-breaking  analysis  to  calculate  a 
realistic  utilization  rate  using  data  from  the  RM&AE,  combined  with  data  from  nonnal  squadron 
operations.  A  DoD-developed  model,  capable  of  projecting  a  utilization  rate  for  a  mature  C-17 
fleet,  showed  that  the  inherent  RM&A  characteristics  of  the  C-17  aircraft  enable  it  to  achieve  a 
wartime  surge  utilization  rate  of  1 5. 1 5  flying  hours  per  aircraft  per  day,  a  key  data  point  used  by 
the  Defense  Acquisition  Board  in  determining  the  number  of  C-17s  needed  for  the  nation’s  airlift 
fleet. 

While  the  GAO  questioned  the  value  of  utilization  rates  in  the  subject  report,  the 
Department  notes  that  in  the  July  1994  GAO  report  (GAO/NSIAD-94-209),  ^‘Airlift 
Requirements-Commercial  Freighters  Can  Help  Meet  Requirements  at  Greatly  Reduced  Cost,” 
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tbe  GAO  indicated  that  utilization  rates  are  a  usehil  statistic  to  compare  aircraft  of  very  different 
capabilities.  The  Department  also  maintains  that  utilization  rates  are  used  as  the  basis  for 
budgeting  for  logistics  resources  (i.e.,  spares,  support  equipment,  manpower,  fuel,  etc.)  required 
to  support  weapon  systems  as  well  as  mission  planning. 

The  DoD  does  not  agree  with  the  GAO  recommendation  to  seek  reimbursement  of 
$750,000  of  the  award  fee  that  was  paid  to  the  contractor  for  the  program’s  performance  during 
the  RM&AE  because  the  formation  personnel  airdrop  and  the  aeromedical  missions  were  not 
flown  during  the  evaluation.  The  Department  maintains  that  the  aircraft  were  correctly  rated  full 
mission  capable  (FMC)  by  maintenance  personnel  in  accordance  with  the  terms  of  the  control 
specifleation.  Per  the  specification,  aircraft  are  considered  FMC  if  all  of  the  equipment  specified 
in  the  Minimum  Equipment  List  is  installed  on  the  aircraft  and  is  operative.  Using  these  criteria, 
the  RM&AE  aircraft  were  rated  FMC  85.1  percent  of  the  time,  exceeding  the  contractual 
requirement  of  72,9  percent  and  entitling  the  contractor  to  the  portion  of  the  award  fee  contested 
by  the  GAO.  The  Department  also  notes  that  three  aeromedical  missions  were  flown  during  the 
RM&AE.  although  no  patients  were  carried.  Additional  comments  on  the  recommendation 
regarding  the  award  fee  can  be  found  at  Enclosure  1. 

In  conclusion,  the  Department  believes  that  the  C- 1 7  RM&AE  provided  a  unique 
opportunity  for  the  Government  to  verify  the  achievement  of  the  aircraft’s  RM&A  specification 
requirements,  and  provided  a  quantitative  basis  for  establishing  the  contractor  s  award  fee 
payments.  The  C-17  RM&AE  was  the  most  thorough,  rigorous  evaluation  of  its  kind  ever 
performed  on  any  aircraft,  and  the  results,  as  noted  in  the  subject  GAO  report,  far  exceeded  the 
RM&AE  requirements,  and  in  fact,  exceeded  the  mature  operational  requirements,  except  for  one 
lower  level  maintainability  parameter.  The  C-17  continues  to  demonstrate  high  RM&A  in 
worldwide  operations,  including  its  support  of  the  U.S.  troops  in  Bosnia, 

The  Department  appreciates  the  opportunity  to  comment  on  the  draft  GAO  report. 


Sincerefy 


iShiyA.  Burt 
yDi^tor,  Test,  Systems 
Engineering  and  Evaluation 

Enclosure 
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Now  on  p.  8. 


GAO  DRAFT  REPORT  -  DATED  MAY  3, 1996 
(GAO  CODE  707111)  OSD  CASE  1147 

«C-17  AIRLIFT:  RM& A  EVALUATION  LESS  DEMANDING 
THAN  INITIALLY  PLANNED” 


DEPARTMENT  OF  DEFENSE  COMMENTS  TO  THE  GAO  RECOMMENDATION 


RECOMMENDATION;  The  GAO  recommended  that  the  Secretary  of  the  Air  Force  initiate 
action  to  recover  the  $750,000  in  incentive  overpayment  from  the  contractor,  (p.  12/GAO  Draft 
Report) 

DoD  RESPONSE;  The  Department  does  not  concur  with  the  GAO’s  assertion  that  the  Air  Force 
overpaid  the  contractor’s  $5.9M  award  fee  for  the  C-17  aircraft  performance  during  the  RM&AE 
by  $750,000  because  formation  personnel  airdrop  capability  was  not  released  and  aeromedical 
evacuation  mission  was  rated  “not  functionally  effective”  by  the  Army  during  operational  test  and 
evaluation.  The  Department  maintains  that  the  contractor  was  paid  an  award  fee  in  accordance 
with  the  existing  contract  specifications.  Aircraft  were  considered  full  mission  capable  (FMC)  per 
the  contract  specification  if  all  of  the  equipment  specified  in  the  Minimum  Equipment  List  was 
installed  on  the  aircraft  and  was  operative.  Since  all  of  the  equipment  specified  in  the  Minimum 
Equipment  List  for  formation  personnel  airdrops  and  aeromedical  evacuation  missions  was 
operative  during  the  C-17  RM&AE,  the  aircraft  were  correctly  rated  FMC  by  maintenance 
personnel.  Hence,  the  Department  asserts  that  the  C-17  contractor  was  entitled  to  the  $750,000 
award  fee  payment  for  the  achievement  of  the  FMC  requirement  in  accordance  with  the  contractual 
terms. 
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Major  Contributors  to  This  Report 


National  Security  and 
International  Affairs 
Division,  Washington, 
D.C. 


Thomas  J.  Denomme 
John  P.  Swain 


Los  Angeles  Field 
Office 


NoelJ.  Lance 
Dorian  R.  Dunbar 
Larry  J.  Bridges 


(7071H) 
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